Efficient Corrections for Standardized Person-Fit Statistics.

Many popular person-fit statistics belong to the class of standardized person-fit statistics, T, and are assumed to have a standard normal null distribution. However, in practice, this assumption is incorrect since T is computed using (a) an estimated ability parameter and (b) a finite number of items. Snijders (Psychometrika 66(3):331-342, 2001) developed mean and variance corrections for T to account for the use of an estimated ability parameter. Bedrick (Psychometrika 62(2):191-199, 1997) and Molenaar and Hoijtink (Psychometrika 55(1):75-106, 1990) developed skewness corrections for T to account for the use of a finite number of items. In this paper, we combine these two lines of research and propose three new corrections for T that simultaneously account for the use of an estimated ability parameter and the use of a finite number of items. The new corrections are efficient in that they only require the analysis of the original data set and do not require the simulation or analysis of any additional data sets. We conducted a detailed simulation study and found that the new corrections are able to control the Type I error rate while also maintaining reasonable levels of power. A real data example is also included.

A two-step item bank calibration strategy based on 1-bit matrix completion for small-scale computerized adaptive testing.

Computerized adaptive testing (CAT) is a widely embraced approach for delivering personalized educational assessments, tailoring each test to the real-time performance of individual examinees. Despite its potential advantages, CAT�s application in small-scale assessments has been limited due to the complexities associated with calibrating the item bank using sparse response data and small sample sizes. This study addresses these challenges by developing a two-step item bank calibration strategy that leverages the 1-bit matrix completion method in conjunction with two distinct incomplete pretesting designs. We introduce two novel 1-bit matrix completion-based imputation methods specifically designed to tackle the issues associated with item calibration in the presence of sparse response data and limited sample sizes. To demonstrate the effectiveness of these approaches, we conduct a comparative assessment against several established item parameter estimation methods capable of handling missing data. This evaluation is carried out through two sets of simulation studies, each featuring different pretesting designs, item bank structures, and sample sizes. Furthermore, we illustrate the practical application of the methods investigated, using empirical data collected from small-scale assessments.

A Model-Based Approach to the Disentanglement and Differential Treatment of Engaged and Disengaged Item Omissions.

Item omissions in large-scale assessments may occur for various reasons, ranging from disengagement to not being capable of solving the item and giving up. Current response-time-based classification approaches allow researchers to implement different treatments of item omissions presumably going back to different mechanisms. These approaches, however, are limited in that they require a clear-cut decision on the underlying missingness mechanism and do not allow to take the uncertainty in classification into account. We present a response-time-based model-based mixture modeling approach that overcomes this limitation. The approach (a) facilitates disentangling item omissions stemming from disengagement from those going back to solution behavior, (b) considers the uncertainty in omission classification, (c) allows for omission mechanisms to vary on the item-by-examinee level, (d) supports investigating person and item characteristics associated with different types of omission behavior, and (e) gives researchers flexibility in deciding on how to handle different types of omissions. The approach exhibits good parameter recovery under realistic research conditions. We illustrate the approach on data from the Programme for the International Assessment of Adult Competencies 2012 and compare it against previous classification approaches for item omissions.

Detecting Differential Item Functioning in Multidimensional Graded Response Models With Recursive Partitioning.

Differential item functioning (DIF) is a common challenge when examining latent traits in large scale surveys. In recent work, methods from the field of machine learning such as model-based recursive partitioning have been proposed to identify subgroups with DIF when little theoretical guidance and many potential subgroups are available. On this basis, we propose and compare recursive partitioning techniques for detecting DIF with a focus on measurement models with multiple latent variables and ordinal response data. We implement tree-based approaches for identifying subgroups that contribute to DIF in multidimensional latent variable modeling and propose a robust, yet scalable extension, inspired by random forests. The proposed techniques are applied and compared with simulations. We show that the proposed methods are able to efficiently detect DIF and allow to extract decision rules that lead to subgroups with well fitting models.

Linking Methods for Multidimensional Forced Choice Tests Using the Multi-Unidimensional Pairwise Preference Model.

Applications of multidimensional forced choice (MFC) testing have increased considerably over the last 20 years. Yet there has been little, if any, research on methods for linking the parameter estimates from different samples. This research addressed that important need by extending four widely used methods for unidimensional linking and comparing the efficacy of new estimation algorithms for MFC linking coefficients based on the Multi-Unidimensional Pairwise Preference model (MUPP). More specifically, we compared the efficacy of multidimensional test characteristic curve (TCC), item characteristic curve (ICC; Haebara, 1980), mean/mean (M/M), and mean/sigma (M/S) methods in a Monte Carlo study that also manipulated test length, test dimensionality, sample size, percentage of anchor items, and linking scenarios. Results indicated that the ICC method outperformed the M/M method, which was better than the M/S method, with the TCC method being the least effective. However, as the number of items "per dimension" and the percentage of anchor items increased, the differences between the ICC, M/M, and M/S methods decreased. Study implications and practical recommendations for MUPP linking, as well as limitations, are discussed.

How Scoring Approaches Impact Estimates of Growth in the Presence of Survey Item Ceiling Effects.

Survey scores are often the basis for understanding how individuals grow psychologically and socio-emotionally. A known problem with many surveys is that the items are all "easy"-that is, individuals tend to use only the top one or two response categories on the Likert scale. Such an issue could be especially problematic, and lead to ceiling effects, when the same survey is administered repeatedly over time. In this study, we conduct simulation and empirical studies to (a) quantify the impact of these ceiling effects on growth estimates when using typical scoring approaches like sum scores and unidimensional item response theory (IRT) models and (b) examine whether approaches to survey design and scoring, including employing various longitudinal multidimensional IRT (MIRT) models, can mitigate any bias in growth estimates. We show that bias is substantial when using typical scoring approaches and that, while lengthening the survey helps somewhat, using a longitudinal MIRT model with plausible values scoring all but alleviates the issue. Results have implications for scoring surveys in growth studies going forward, as well as understanding how Likert item ceiling effects may be contributing to replication failures.

Development and validation of health-oriented personal evaluation for the community-dwelling older person based on the International Classification of Functioning, Disability and Health.

BACKGROUND: The International Classification of Functioning, Disability and Health (ICF) offers a standardized international terminology to operationalize function management across multiple domains, but the summary score of the ICF qualifier scale provides limited information on the comparison of personal abilities and functioning difficulties. OBJECTIVES: To enhance the interpretative power of the ICF-based Health-oriented Personal Evaluation for the community-dwelling older person (iHOPE-OP) scale through the implementation of the item response theory (IRT) modelling. METHODS: This cross-sectional, multi-centre study administrated 161 ICF categories (58 on body functions, 15 on body structures, 60 on activities or participation and 28 on environmental factors) to evaluate the functional level of 338 older citizens (female = 158, male = 180) residing in community or supportive living facilities. The validation process encompassed assessing the IRT model fitness and evaluating the psychometric properties of the IRT-derived iHOPE-OP scale. RESULTS: The age of participants ranged from 60 to 94.57, with the mean age of 70. The analysis of non-parametric and parametric models revealed that the three-parameter logistic IRT model, with a dichotomous scoring principle, exhibited the best fit. The 53-item iHOPE-OP scale demonstrated high reliability (Cronbach's α = 0.9729, Guttman's lambda-2 = 0.9749, Molenaar-Sijtsma Statistic = 0.9803, latent class reliability coefficient = 0.9882). There was a good validity between person abilities and the Barthel Index (p < .001, r = .83), as well as instrumental activities of daily living (p < .001, r = .84). CONCLUSIONS: IRT methods generate the reliable and valid iHOPE-OP scale with the most discriminable and minimal items to represent the older person's functional performance at a comprehensive level. The use of the Wright map can aid in presby-functioning management by visualizing item difficulties and person abilities. IMPLICATIONS FOR PRACTICE: Considering the intricate and heterogeneous health status of older persons, a single functional assessment tool might not fulfil the need to fully understand the multifaceted health status. For use in conjunction with the IRT and ICF framework, the reliable and valid iHOPE-OP scale was developed and can be applied to capture presby-functioning. The Wright map depicts the distribution of item difficulties and person abilities on the same scale that facilitates person-centred goal setting and tailors intervention.

Development and validation of the patient-reported outcome for older people living with HIV/AIDS in China (PROHIV-OLD).

BACKGROUND: The involvement of quality of life as the UNAIDS fourth 90 target to monitor the global HIV response highlighted the development of patient-reported outcome (PRO) measures to help address the holistic needs of people living with HIV/AIDS (PLWHA) beyond viral suppression. This study developed and tested preliminary measurement properties of a new patient-reported outcome (PROHIV-OLD) measure designed specifically to capture influences of HIV on patients aged 50 and older in China. METHODS: Ninety-three older people living with HIV/AIDS (PLWHA) were interviewed to solicit items and two rounds of patient cognitive interviews were conducted to modify the content and wording of the initial items. A validation study was then conducted to refine the initial instrument and evaluate measurement properties. Patients were recruited between February 2021 and November 2021, and followed six months later after the first investigation. Classical test theory (CTT) and item response theory (IRT) were used to select items using the baseline data. The follow-up data were used to evaluate the measurement properties of the final instrument. RESULTS: A total of 600 patients were recruited at the baseline. Of the 485 patients who completed the follow-up investigation, 483 were included in the validation sample. The final scale of PROHIV-OLD contained 25 items describing five dimensions (physical symptoms, mental status, illness perception, family relationship, and treatment). All the PROHIV-OLD dimensions had satisfactory reliability with Cronbach's alpha coefficient, McDonald's ω, and composite reliability of each dimension being all higher than 0.85. Most dimensions met the test-retest reliability standard except for the physical symptoms dimension (ICC = 0.64). Confirmatory factor analysis supported the structural validity of the final scale, and the model fit index satisfied the criterion. The correlations between dimensions of PROHIV-OLD and MOS-HIV met hypotheses in general. Significant differences on scores of the PROHIV-OLD were found between demographic and clinical subgroups, supporting known-groups validity. CONCLUSIONS: The PROHIV-OLD was found to have good feasibility, reliability and validity for evaluating health outcome of Chinese older PLWHA. Other measurement properties such as responsiveness and interpretability will be further examined.

Reducing Attenuation Bias in Regression Analyses Involving Rating Scale Data via Psychometric Modeling.

Many studies in fields such as psychology and educational sciences obtain information about attributes of subjects through observational studies, in which raters score subjects using multiple-item rating scales. Error variance due to measurement effects, such as items and raters, attenuate the regression coefficients and lower the power of (hierarchical) linear models. A modeling procedure is discussed to reduce the attenuation. The procedure consists of (1) an item response theory (IRT) model to map the discrete item responses to a continuous latent scale and (2) a generalizability theory (GT) model to separate the variance in the latent measurement into variance components of interest and nuisance variance components. It will be shown how measurements obtained from this mixture of IRT and GT models can be embedded in (hierarchical) linear models, both as predictor or criterion variables, such that error variance due to nuisance effects are partialled out. Using examples from the field of educational measurement, it is shown how general-purpose software can be used to implement the modeling procedure.

Novel indices reveal that pollinator exposure to pesticides varies across biological compartments and crop surroundings.

Declines in insect pollinators have been linked to a range of causative factors such as disease, loss of habitats, the quality and availability of food, and exposure to pesticides. Here, we analysed an extensive dataset generated from pesticide screening of foraging insects, pollen-nectar stores/beebread, pollen and ingested nectar across three species of bees collected at 128 European sites set in two types of crop. In this paper, we aimed to (i) derive a new index to summarise key aspects of complex pesticide exposure data and (ii) understand the links between pesticide exposures depicted by the different matrices, bee species and apple orchards versus oilseed rape crops. We found that summary indices were highly correlated with the number of pesticides detected in the related matrix but not with which pesticides were present. Matrices collected from apple orchards generally contained a higher number of pesticides (7.6 pesticides per site) than matrices from sites collected from oilseed rape crops (3.5 pesticides), with fungicides being highly represented in apple crops. A greater number of pesticides were found in pollen-nectar stores/beebread and pollen matrices compared with nectar and bee body matrices. Our results show that for a complete assessment of pollinator pesticide exposure, it is necessary to consider several different exposure routes and multiple species of bees across different agricultural systems.

Unveiling disparities: examining differential item functioning's impact on racial health equity among white and black populations.

OBJECTIVES: This paper aims to examine the psychometric properties of social capital indicators, comparing Black and White respondents to identify the extent of measurement invariance in social capital by race. STUDY DESIGN: We used data from the longitudinal study Midlife in the United States (MIDUS), waves 1 through 3 (1995-2016). METHODS: Data were from 6513 respondents (5604 White and 909 Black respondents). Social capital indicators were social cohesion, contributions to community, and community involvement. We used Structural Equation Modeling and Item Response Theory methods to test for measurement invariance in social capital by race. RESULTS: We observed violations of longitudinal and multi-group measurement invariance (MI) at configural and metric levels on two scales. Factor structures and indicator loadings were inconsistent over time. In IRT analysis, 'Many people come for advice' exhibited Differential Item Functioning (DIF), indicating a consistent advantage for White respondents on the contributions to community scale. Despite similar social capital levels (P(χ2,2) = 0.00), DIF was found in all contributions to community items and some community involvement items when examining race and education interaction. CONCLUSIONS: Invariance issues in social capital items suggest potential biases in comparing Black and White respondents. Recognizing these biases is essential. Future social capital research should assess existing data assumptions and involve stakeholders from diverse communities in creating new items.

Understanding Vietnamese chicken farmers' knowledge and practices related to antimicrobial resistance using an item response theory approach.

Introduction: Antimicrobial resistance (AMR) poses a threat to animal and human health, as well as food security and nutrition. Development of AMR is accelerated by over- and misuse of antimicrobials as seen in many livestock systems, including poultry production. In Vietnam, high AMR levels have been reported previously within poultry production, a sector which is dominated by small-scale farming, even though it is intensifying. This study focuses on understanding small- and medium-scale chicken farmers' knowledge and practices related to AMR by applying an item response theory (IRT) approach, which has several advantages over simpler statistical methods. Methods: Farmers representing 305 farms in Thai Nguyen province were interviewed from November 2021 to January 2022, using a structured questionnaire. Results generated with IRT were used in regression models to find associations between farm characteristics, and knowledge and practice levels. Results: Descriptive results showed that almost all farmers could buy veterinary drugs without prescription in the local community, that only one third of the farmers received veterinary professional advice or services, and that the majority of farmers gave antibiotics as a disease preventive measure. Regression analysis showed that multiple farm characteristics were significantly associated to farmers' knowledge and practice scores. Conclusion: The study highlights the complexity when tailoring interventions to move towards more medically rational antibiotic use at farms in a setting with high access to over-the-counter veterinary drugs and low access to veterinary services, since many on-farm factors relevant for the specific context need to be considered.

A neural network paradigm for modeling psychometric data and estimating IRT model parameters: Cross estimation network.

This paper presents a novel approach known as the cross estimation network (CEN) for fitting the datasets obtained from psychological or educational tests and estimating the parameters of item response theory (IRT) models. The CEN is comprised of two subnetworks: the person network (PN) and the item network (IN). The PN processes the response pattern of individual respondent and generates an estimate of the underlying ability, while the IN takes in the response pattern of individual item and outputs the estimates of the item parameters. Four simulation studies and an empirical study were comprehensively and rigorously conducted to investigate the performance of CEN on parameter estimation of the two-parameter logistic model under various testing scenarios. Results showed that CEN effectively fit the training data and produced accurate estimates of both person and item parameters. The trained PN and IN adhered to AI principles and acted as intelligent agents, delivering commendable evaluations for even unseen patterns of new respondents and items.

Measuring the variability of personality traits with interval responses: Psychometric properties of the dual-range slider response format : Measuring variability with interval responses.

Measuring the variability in persons' behaviors and experiences using ecological momentary assessment is time-consuming and costly. We investigate whether interval responses provided through a dual-range slider (DRS) response format can be used as a simple and efficient alternative: Respondents indicate variability in their behavior in a retrospective rating by choosing a lower and an upper bound on a continuous, bounded scale. We investigate the psychometric properties of this response format as a prerequisite for further validation. First, we assess the test-retest reliability of factor-score estimates for the width of DRS intervals. Second, we test whether factor-score estimates of the visual analog scale (VAS) and the location of DRS intervals show convergent validity. Third, we investigate whether factor-score estimates for the DRS are uncorrelated between different personality scales. We present a longitudinal multitrait-multimethod study using two personality scales (Extraversion, Conscientiousness) and two response formats (VAS, DRS) at two measurement occasions (6-8 weeks apart) for which we estimate factor-score correlations in a joint item response theory model. The test-retest reliability of the width of DRS intervals was high ( ρ ^ ≥ . 73 ). Also, convergent validity between location scores of VAS and DRS was high ( ρ ^ ≥ . 88 ). Conversely, discriminant validity of the width of DRS intervals between Extraversion and Conscientiousness was poor ( ρ ^ ≥ . 94 ). In conclusion, the DRS seems to be a reliable response format that could be used to measure the central tendency of a trait equivalently to the VAS. However, it might not be well suited for measuring intra-individual variability in personality traits.

Measurement Invariance in Intellectual and Developmental Disability Research.

Measurement invariance (MI) is a psychometric property of an instrument indicating the degree to which scores from an instrument are comparable across groups. In recent years, there has been a marked uptick in publications using MI in intellectual and developmental disability (IDD) samples. Our goal here is to provide an overview of why MI is important to IDD researchers and to describe some challenges to evaluating it, with an eye towards nudging our subfield into a more thoughtful and measured interpretation of studies using MI.

Extended Asymptotic Identifiability of Nonparametric Item Response Models.

Nonparametric item response models provide a flexible framework in psychological and educational measurements. Douglas (Psychometrika 66(4):531-540, 2001) established asymptotic identifiability for a class of models with nonparametric response functions for long assessments. Nevertheless, the model class examined in Douglas (2001) excludes several popular parametric item response models. This limitation can hinder the applications in which nonparametric and parametric models are compared, such as evaluating model goodness-of-fit. To address this issue, We consider an extended nonparametric model class that encompasses most parametric models and establish asymptotic identifiability. The results bridge the parametric and nonparametric item response models and provide a solid theoretical foundation for the applications of nonparametric item response models for assessments with many items.

Development and validation of the newly developed Preschool Theory of Mind Assessment (ToMA-P).

Introduction: Theory of mind (ToM) refers to the ability to understand and attribute mental states to oneself and others. A ToM measure is warranted for preschool children to assess their ToM development from a multidimensional perspective (i.e., cognitive and affective dimensions). This study aimed to develop the Preschool Theory of Mind Assessment (ToMA-P) and to evaluate its construct validity and applicability. Methods: The ToMA-P was developed based on comprehensive literature review and revised with expert panel feedback. Its psychometric properties were evaluated in 205 typically developing preschoolers with Rasch analysis for its dimensionality, item difficulties, and convergent validity. Results: The results indicated that all ToMA-P items, except for one, fit the hypothesized two-dimensional construct. The item difficulties in the cognitive and affective dimensions followed developmental sequences. The ToMA-P scores exhibited good convergent validity, as evidenced by its significant correlations with age, verbal comprehension, adaptive functions, and daily ToM performance (p < 0.05). Children's responses and behaviors also showed that the ToMA-P has good applicability. Discussion: This study provides empirical evidence that the ToMA-P measures cognitive and affective ToM following developmental sequences, and that it has potential as a clinical tool for assessing ToM in preschool children.

Three new corrections for standardized person-fit statistics for tests with polytomous items.

Recent years have seen a growing interest in the development of person-fit statistics for tests with polytomous items. Some of the most popular person-fit statistics for such tests belong to the class of standardized person-fit statistics, T $$ T $$ , that is assumed to have a standard normal null distribution. However, this distribution only holds when (a) the true ability parameter is known and (b) an infinite number of items are available. In practice, both conditions are violated, and the quality of person-fit results is expected to deteriorate. In this paper, we propose three new corrections for T $$ T $$ that simultaneously account for the use of an estimated ability parameter and the use of a finite number of items. The three new corrections are direct extensions of those that were developed by Gorney et al. (Psychometrika, 2024, https://doi.org/10.1007/s11336-024-09960-x) for tests with only dichotomous items. Our simulation study reveals that the three new corrections tend to outperform not only the original statistic T $$ T $$ but also an existing correction for T $$ T $$ proposed by Sinharay (Psychometrika, 2016, 81, 992). Therefore, the new corrections appear to be promising tools for assessing person fit in tests with polytomous items.

Recognize the Value of the Sum Score, Psychometrics' Greatest Accomplishment.

The sum score on a psychological test is, and should continue to be, a tool central in psychometric practice. This position runs counter to several psychometricians' belief that the sum score represents a pre-scientific conception that must be abandoned from psychometrics in favor of latent variables. First, we reiterate that the sum score stochastically orders the latent variable in a wide variety of much-used item response models. In fact, item response theory provides a mathematically based justification for the ordinal use of the sum score. Second, because discussions about the sum score often involve its reliability and estimation methods as well, we show that, based on very general assumptions, classical test theory provides a family of lower bounds several of which are close to the true reliability under reasonable conditions. Finally, we argue that eventually sum scores derive their value from the degree to which they enable predicting practically relevant events and behaviors. None of our discussion is meant to discredit modern measurement models; they have their own merits unattainable for classical test theory, but the latter model provides impressive contributions to psychometrics based on very few assumptions that seem to have become obscured in the past few decades. Their generality and practical usefulness add to the accomplishments of more recent approaches.

Can you tell people's cognitive ability level from their response patterns in questionnaires?

Questionnaires are ever present in survey research. In this study, we examined whether an indirect indicator of general cognitive ability could be developed based on response patterns in questionnaires. We drew on two established phenomena characterizing connections between cognitive ability and people's performance on basic cognitive tasks, and examined whether they apply to questionnaires responses. (1) The worst performance rule (WPR) states that people's worst performance on multiple sequential tasks is more indicative of their cognitive ability than their average or best performance. (2) The task complexity hypothesis (TCH) suggests that relationships between cognitive ability and performance increase with task complexity. We conceptualized items of a questionnaire as a series of cognitively demanding tasks. A graded response model was used to estimate respondents' performance for each item based on the difference between the observed and model-predicted response ("response error" scores). Analyzing data from 102 items (21 questionnaires) collected from a large-scale nationally representative sample of people aged 50+ years, we found robust associations of cognitive ability with a person's largest but not with their smallest response error scores (supporting the WPR), and stronger associations of cognitive ability with response errors for more complex than for less complex questions (supporting the TCH). Results replicated across two independent samples and six assessment waves. A latent variable of response errors estimated for the most complex items correlated .50 with a latent cognitive ability factor, suggesting that response patterns can be utilized to extract a rough indicator of general cognitive ability in survey research.